Add Spark 4.1 support - #287
Conversation
billpratt
changed the title
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Generated by AI (model: us.anthropic.claude-opus-4-8, prompt: 295226dd) — may not be fully accurate. Reply if this doesn't help.
Reviewed by Shadow · github.com/sudsali/shadow
Additional feedback:
tests/test_checks.py:677 — DESIGN: The one-ULP-tolerance rationale applies to all mutual-information results, but only one of three MI assertions in test_hasMutualInformation was loosened; the remaining x == 1.0986122886681096 comparisons at lines 677 and 682 may still be flaky on Spark 4.1.
tests/test_checks.py:677
lambda x: x == 1.0986122886681096, line 682 same pattern, line 686 (loosened by diff tomath.isclose(x, 0.7324081924454064, rel_tol=0.0, abs_tol=1e-15)). The diff comment states 'Spark 3.5 and 4.1 can differ by one ULP in mutual information results' — this rationale applies equally to all three MI assertions.
Refutation trail (why this survived the Critic's disprove pass)
Hypothesis (Investigator): The incremental diff loosens only the ("c", "b") mutual-information assertion to a tolerance-based check, but the sibling exact-equality assertions in the same test (x == 1.0986122886681096 at lines 675-684) are left as strict ==, so they remain vulnerable to the same one-ULP Spark 4.1 divergence the comment cites.
Disprove attempt (Critic): Read tests/test_checks.py lines 674-688 at PR-head SHA. The test_hasMutualInformation body contains three assertions: two comparing against 1.0986122886681096 via lambda x: x == ... and one against 0.7324081924454064. The incremental diff only wraps the third (0.7324...) in math.isclose(..., abs_tol=1e-15); the two 1.0986... comparisons remain exact ==. If Spark 4.1 can differ by one ULP for MI generally (per the added comment), those two assertions can still flake on Spark 4.1.
The Critic's default verdict is OVERTURNED. UPHELD findings are those it tried — and failed — to refute.
0abc8d6 to
09963d4
Compare
Copilot-Session: 76955bb1-ca95-47f6-996d-0d1f505cc5f9
Allow the one-ULP mutual information rounding difference between Spark 3.5 and Spark 4.1. Co-authored-by: Copilot App <[email protected]>
Materialize converted iterables as List so Deequ receives a compatible immutable Seq on Scala 2.12 and 2.13. Co-authored-by: Copilot App <[email protected]>
Keep Python 3.9 on NumPy 1.x and Pandas 1.x so Spark 3.x remains compatible while Python 3.10 uses the Spark 4.1 dependency floor. Co-authored-by: Copilot App <[email protected]>
2392349 to
3d43dd4
Compare
Refresh the Poetry content hash after resolving the upstream dependency constraint merge. Co-authored-by: Copilot App <[email protected]>
|
@sudsali this is ready for review. Could a spark 4.1 version of deequ 2.0.21 be published? That way, the versions match between spark 3.5 and 4.1 |
1 participant
Summary
Spark 4.1 support, tracked by #286.
com.amazon.deequ:deequ:2.0.18-spark-4.1Coordination
This draft is intended to coordinate with #283. Its final form should adopt #283's Spark 3.5 baseline and resolve overlapping dependency, CI, and documentation changes after that PR merges.
Follow-up before merge
poetry.lockfrom a public-PyPI-capable environmentValidation
A local full test run against PySpark 4.1.2, Java 17, Scala 2.13, and
deequ:2.0.18-spark-4.1completed successfully.